Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: follower node sync from DA #631

Closed
wants to merge 73 commits into from
Closed

feat: follower node sync from DA #631

wants to merge 73 commits into from

Conversation

NazariiDenha
Copy link

@NazariiDenha NazariiDenha commented Feb 6, 2024 •
edited by jonastheis
Loading

1. Purpose or design rationale of this PR

This PR implements a follower node from DA/L1 mode which reproduces the L2 state solely from L1 events and loading data from DA (calldata is retrieved from L1 RPC directly, historical blobs are loaded via beacon node or blob APIs and verified via versionedHash).

On a high level, it works as follows: the L2 functionality of the node is disabled and instead it connects only to the configured L1 RPC, retrieves all rollup events (commit batch, revert, finalize), L1 messages and batch data (i.e. calldata or blobs since Bernoulli). Once an event is finalized on L1 the resulting state (meaning L2 state and blocks) are derived and verified from this data.

The derivation process works by implementing a pipeline with following steps:
- DAQueue: uses DataSource to retrieve events and corresponding data (calldata or blob).
- BatchQueue: sorts different DATypes and returns committed, finalized batches in order.
- BlockQueue: converts batches to PartialBlocks that can be used to create the L2 state
- DASyncer: executes PartialBlock and inserts into chain

How to run?

Run l2geth with the --da.sync flag and provide blob APIs with e.g. --da.blob.blobscan=https://api.blobscan.com/blobs/ --da.blob.blocknative=https://api.ethernow.xyz/v1/blob/ for mainnet and --da.blob.blobscan=https://api.sepolia.blobscan.com/blobs/ for Sepolia.

./build/bin/geth --scroll \
  --datadir "tmp/mainnet-l2geth-datadir" \
  --gcmode archive --cache.noprefetch \
  --http --http.addr "0.0.0.0" --http.port 8545 --http.api "eth,net,web3,debug,scroll" \
  --l1.endpoint "<L1 RPC node>" \
  --da.sync=true \
  --da.blob.blobscan=https://api.blobscan.com/blobs/ --da.blob.blocknative=https://api.ethernow.xyz/v1/blob/ \
  --verbosity 3

A full sync will take about ~2 days depending on the speed of the RPC node and the local machine. Progess is reported as follows for every 1000 blocks applied:

INFO [08-01|16:44:42.173] L1 sync progress                         blockhain height=87000 block hash=608eec..880ebd root=218215..9a58a2

Limitations

The state root of a block can be reproduced when using this mode of syncing but currently not the block hash. This is due to the fact that currently the header fields difficulty and extraData are not stored on DA but these fields are utilized by Clique consensus which is used by the Scroll protocol. This will be fixed in a future upgrade where the main implementation on l2geth is already done: #903 #913.

To verify the locally created state root against mainnet, we can do the following:

# query local block info
curl localhost:8545 -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getHeaderByNumber","params":["0x2AF8"],"id":0}' | jq

# query mainnet block info
curl https://rpc.scroll.io -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getHeaderByNumber","params":["0x2AF8"],"id":0}' | jq

By comparing the headers we can most importantly see that state root , receiptsRoot and everything that has to do with the state matches. However, the following fields will be different:

  • difficulty and therefore totalDifficulty
  • extraData
  • size due to differences in header size
  • hash and therefore parentHash

Example local output for block 11000:

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "difficulty": "0xa",
    "extraData": "0x0102030405060708",
    "gasLimit": "0x989680",
    "gasUsed": "0xa410",
    "hash": "0xf3cdafbe35d5e7c18d8274bddad9dd12c94b83a81cefeb82ebb73fa799ff9fcc",
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "miner": "0x0000000000000000000000000000000000000000",
    "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "nonce": "0x0000000000000000",
    "number": "0x2af8",
    "parentHash": "0xde244f7e8bc54c8809e6c2ce65c439b58e90baf11f6cf9aaf8df33a827bd01ab",
    "receiptsRoot": "0xd95b673818fa493deec414e01e610d97ee287c9421c8eff4102b1647c1a184e4",
    "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "size": "0x252",
    "stateRoot": "0x0f387e78e4a7457a318c7bce7cde0b05c3609347190144a7e105ef05194ae218",
    "timestamp": "0x6526db8e",
    "totalDifficulty": "0x1adb1",
    "transactionsRoot": "0x6a81c9342456693d57963883983bba024916f4d277392c9c1dc497e3518a78e3"
  }
}

Example remote output:

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
    "difficulty": "0x2",
    "extraData": "0xd883050000846765746888676f312e31392e31856c696e7578000000000000009920319c246ec8ae4d4f73f07d79f68b2890e9c2033966efe5a81aedddae12875c3170f0552f48b7e5d8e92ac828a6008b2ba7c5b9c4a0af1692337bbdc792be01",
    "gasLimit": "0x989680",
    "gasUsed": "0xa410",
    "hash": "0xb7848d5b300247d7c33aeba0f1b33375e1cb3113b950dffc140945e9d3d88d58",
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "miner": "0x0000000000000000000000000000000000000000",
    "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "nonce": "0x0000000000000000",
    "number": "0x2af8",
    "parentHash": "0xa93e6143ab213a044eb834cdd391a6ef2c818de25b04a3839ee44a75bd28a2c7",
    "receiptsRoot": "0xd95b673818fa493deec414e01e610d97ee287c9421c8eff4102b1647c1a184e4",
    "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "size": "0x2ab",
    "stateRoot": "0x0f387e78e4a7457a318c7bce7cde0b05c3609347190144a7e105ef05194ae218",
    "timestamp": "0x6526db8e",
    "totalDifficulty": "0x55f1",
    "transactionsRoot": "0x6a81c9342456693d57963883983bba024916f4d277392c9c1dc497e3518a78e3"
  }
}

2. PR title

Your PR title must follow conventional commits (as we are doing squash merge for each PR), so it must start with one of the following types:

  • build: Changes that affect the build system or external dependencies (example scopes: yarn, eslint, typescript)
  • ci: Changes to our CI configuration files and scripts (example scopes: vercel, github, cypress)
  • docs: Documentation-only changes
  • feat: A new feature
  • fix: A bug fix
  • perf: A code change that improves performance
  • refactor: A code change that doesn't fix a bug, or add a feature, or improves performance
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
  • test: Adding missing tests or correcting existing tests

3. Deployment tag versioning

Has the version in params/version.go been updated?

  • This PR doesn't involve a new deployment, git tag, docker image tag, and it doesn't affect traces
  • Yes

4. Breaking change label

Does this PR have the breaking-change label?

  • This PR is not a breaking change
  • Yes

@NazariiDenha NazariiDenha marked this pull request as draft February 6, 2024 09:03
@NazariiDenha NazariiDenha marked this pull request as ready for review February 13, 2024 09:07
@NazariiDenha NazariiDenha changed the title implement l1rpcfetcher and prepare for implementing da_syncer implement l1rpcfetcher and da_syncer Feb 13, 2024
@Thegaram Thegaram requested review from vyzo and Thegaram February 16, 2024 09:40
vyzo
vyzo reviewed Feb 17, 2024
View reviewed changes
Copy link

@vyzo vyzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first pass generally looks ok, left some comments.

I will do another pass later.

cmd/utils/flags.go Outdated Show resolved Hide resolved
core/rawdb/accessors_da_syncer.go Outdated Show resolved Hide resolved
core/rawdb/accessors_da_syncer.go Outdated Show resolved Hide resolved
rollup/da_syncer/da_syncer.go Outdated Show resolved Hide resolved
rollup/da_syncer/l1rpc_fetcher.go Outdated Show resolved Hide resolved
rollup/da_syncer/l1rpc_fetcher.go Outdated Show resolved Hide resolved
@NazariiDenha NazariiDenha changed the title implement l1rpcfetcher and da_syncer DON'T MERGE: follower node sync from DA(L1) Apr 30, 2024
NazariiDenha added 12 commits May 17, 2024 17:50
@NazariiDenha
refactor code to new design with pipeline
5f11287
@NazariiDenha
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
885bd3e 
…t/sync-directly-from-da
@NazariiDenha
adapt for codecv0 types
6b9e837
@NazariiDenha
prepare for blob dta asource
c6201c3
@NazariiDenha
implement blob fethcer
27a4e51
@NazariiDenha
implement blob fethcer
6991da0
@NazariiDenha
fixes after testing and add synced l1 height to db
e483219
@NazariiDenha
Merge branch 'feat/sync-directly-from-da' of github.com:scroll-tech/g…
48dd5d8 
…o-ethereum into feat/sync-directly-from-da
@NazariiDenha
fixes and improvements after testing
eb377fe
@NazariiDenha
fixes and improvements after testing
5a48000
@NazariiDenha
Merge branch 'feat/sync-directly-from-da' of github.com:scroll-tech/g…
01e5f9c 
…o-ethereum into feat/sync-directly-from-da
@NazariiDenha
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
1a84506 
…t/sync-directly-from-da
jonastheis
jonastheis previously approved these changes Aug 8, 2024
View reviewed changes
@jonastheis jonastheis force-pushed the feat/sync-directly-from-da branch from 8f1f6b7 to 399dcfc Compare August 13, 2024 04:35
0xmountaintop
0xmountaintop previously approved these changes Aug 14, 2024
View reviewed changes
0xmountaintop
0xmountaintop previously approved these changes Aug 17, 2024
View reviewed changes
@NazariiDenha @jonastheis
(follower_node)support beacon node client as blob provider (#988)
9e3c838 
* support beacon node client as blob provider

* fix

* fix formatting

* use url.JoinPath instead of path

* don't move pos each time in blob_client_list

---------

Co-authored-by: jonastheis <[email protected]>
jonastheis added a commit that referenced this pull request Aug 29, 2024
@jonastheis
feat: follower node sync from DA #631
00494d5 
 Conflicts:
	cmd/geth/main.go
	core/state_processor_test.go
	core/txpool/legacypool/legacypool.go
	eth/backend.go
	eth/ethconfig/config.go
	eth/gasprice/gasprice_test.go
	eth/handler.go
	eth/protocols/eth/broadcast.go
	eth/protocols/eth/handlers.go
	go.mod
	go.sum
	miner/miner.go
	miner/miner_test.go
	miner/scroll_worker.go
	miner/scroll_worker_test.go
	params/config.go
	params/version.go
	rollup/rollup_sync_service/rollup_sync_service_test.go
@jonastheis jonastheis mentioned this pull request Aug 29, 2024
Merged
13 tasks
colinlyguo
colinlyguo reviewed Sep 11, 2024
View reviewed changes
cmd/utils/flags.go Show resolved Hide resolved
miner/scroll_worker.go Show resolved Hide resolved
core/blockchain.go Show resolved Hide resolved
@jonastheis
Copy link

Closing this PR as it is superseded by #1013 which is based on the new base branch syncupstream/active.

@jonastheis jonastheis closed this Oct 14, 2024
jonastheis added a commit that referenced this pull request Oct 16, 2024
@jonastheis @NazariiDenha @0xmountaintop
feat: follower node sync from DA rebased to syncUpstream/active (#1013
8454491 
)

* feat: follower node sync from DA #631

 Conflicts:
	cmd/geth/main.go
	core/state_processor_test.go
	core/txpool/legacypool/legacypool.go
	eth/backend.go
	eth/ethconfig/config.go
	eth/gasprice/gasprice_test.go
	eth/handler.go
	eth/protocols/eth/broadcast.go
	eth/protocols/eth/handlers.go
	go.mod
	go.sum
	miner/miner.go
	miner/miner_test.go
	miner/scroll_worker.go
	miner/scroll_worker_test.go
	params/config.go
	params/version.go
	rollup/rollup_sync_service/rollup_sync_service_test.go

* fixes after merge

* fix: panic after startup due to misconfigured api

* fix bug of not calculating block state root

* minor adjustment of fix

* address review comments

* upgrade da-codec to scroll-tech/da-codec@41c6486

* fix go.mod after merge

* fix some tests

---------

Co-authored-by: Nazarii Denha <[email protected]>
Co-authored-by: 0xmountaintop <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@colinlyguo colinlyguo colinlyguo left review comments

@semgrep-app semgrep-app[bot] semgrep-app[bot] left review comments

@jonastheis jonastheis jonastheis left review comments

@vyzo vyzo vyzo left review comments

@0xmountaintop 0xmountaintop 0xmountaintop left review comments

@Thegaram Thegaram Awaiting requested review from Thegaram

Assignees
No one assigned
Labels
bump-version
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@NazariiDenha @jonastheis @vyzo @0xmountaintop @colinlyguo